Consistency of Nearest Neighbor Methods

In this lecture we return to the study of consistency properties of learning algorithms, where we will be interested in the question of whether the generalization error of the function learned by an algorithm approaches the Bayes error in the limit of infinite data. In particular, we will consider consistency properties of the simple k-nearest neighbor (k-NN) classification algorithm (in the next lecture, we will investigate consistency properties of algorithms such as SVMs and AdaBoost that effectively minimize a convex upper bound on the zero-one loss). We start with a brief review of statistical consistency and related results.